A. Field of the Invention
Concepts described herein relate generally to data compression, and more particularly, to a compression scheme with disk-based dictionary in a wide area network optimization device.
B. Description of Related Art
Geographically distributed entities, such as a business with multiple local offices in different locations, frequently desire to maintain a logically unified proprietary network that is accessible by all of the geographically distributed entities. To accomplish this, a wide area network (WAN) may be used to link the different locations.
Information technology managers within geographically distributed entities face the conflicting requirements of keeping costs in check while providing users with new and faster applications. This conflict is most acute on the WAN, where costs can be high and getting applications to perform well is hampered by limited bandwidth and high latency.
Solutions are known to help improve performance over the WAN portion of a network. These solutions may be based on a number of technologies, including data compression, data prefetching and caching, application-specific acceleration, and policy-based bandwidth allocation.
Data caching and compression, in particular, can be used to reduce the perceived latency (response time) of data accesses over a WAN. Compression and caching techniques may be disk-based, as opposed to semiconductor memory based, due to orders-of-magnitude larger storage capacity of typical hard disk drives.
Compression techniques using disk-based dictionary can be particularly effective in compressing large file transfers. With this type of compression technique, data may be stored on disk drives at compression devices at opposite ends of the WAN. When a block of data that has been previously sent is again received by the compression device, the compression device does not need to send the actual block of data. Instead, it may simply send a reference to the block of data to the device on the other end of the WAN.
In order to perform dictionary-based compression efficiently, some form of index is usually constructed and maintained by the compression devices. Efficiently constructing and using such an index can be important to maximizing the effectiveness of the compression device.